I write software for brain imaging: http://nipy.org
I teach students how to use code for neuroscience.
The students:
numpy
, scipy
, matplotlib
and my code.pip install matplotlib
anybody?And I:
numpy
, scipy
, matplotlib
, sympy
.Megainstallers? But - too many systems to test against, no control over packaging.
Six months ago I was in despair.
What about all the dependencies? It's all too much.
In January - OSX wheels were allowed on pypi.
My life has changed.
These guys are so incredibly easy to use...
%%bash
pip install wheel
cd markupsafe
rm -rf build # just to be safe
python setup.py bdist_wheel
Downloading/unpacking wheel Installing collected packages: wheel Successfully installed wheel Cleaning up... running bdist_wheel running build running build_py creating build creating build/lib.macosx-10.6-intel-2.7 creating build/lib.macosx-10.6-intel-2.7/markupsafe copying markupsafe/__init__.py -> build/lib.macosx-10.6-intel-2.7/markupsafe copying markupsafe/_compat.py -> build/lib.macosx-10.6-intel-2.7/markupsafe copying markupsafe/_constants.py -> build/lib.macosx-10.6-intel-2.7/markupsafe copying markupsafe/_native.py -> build/lib.macosx-10.6-intel-2.7/markupsafe copying markupsafe/tests.py -> build/lib.macosx-10.6-intel-2.7/markupsafe running egg_info creating MarkupSafe.egg-info writing dependency_links to MarkupSafe.egg-info/dependency_links.txt writing MarkupSafe.egg-info/PKG-INFO writing top-level names to MarkupSafe.egg-info/top_level.txt writing manifest file 'MarkupSafe.egg-info/SOURCES.txt' reading manifest file 'MarkupSafe.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'MarkupSafe.egg-info/SOURCES.txt' copying markupsafe/_speedups.c -> build/lib.macosx-10.6-intel-2.7/markupsafe running build_ext building 'markupsafe._speedups' extension creating build/temp.macosx-10.6-intel-2.7 creating build/temp.macosx-10.6-intel-2.7/markupsafe gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -arch i386 -arch x86_64 -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c markupsafe/_speedups.c -o build/temp.macosx-10.6-intel-2.7/markupsafe/_speedups.o gcc-4.2 -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g build/temp.macosx-10.6-intel-2.7/markupsafe/_speedups.o -o build/lib.macosx-10.6-intel-2.7/markupsafe/_speedups.so installing to build/bdist.macosx-10.6-intel/wheel running install running install_lib creating build/bdist.macosx-10.6-intel creating build/bdist.macosx-10.6-intel/wheel creating build/bdist.macosx-10.6-intel/wheel/markupsafe copying build/lib.macosx-10.6-intel-2.7/markupsafe/__init__.py -> build/bdist.macosx-10.6-intel/wheel/markupsafe copying build/lib.macosx-10.6-intel-2.7/markupsafe/_compat.py -> build/bdist.macosx-10.6-intel/wheel/markupsafe copying build/lib.macosx-10.6-intel-2.7/markupsafe/_constants.py -> build/bdist.macosx-10.6-intel/wheel/markupsafe copying build/lib.macosx-10.6-intel-2.7/markupsafe/_native.py -> build/bdist.macosx-10.6-intel/wheel/markupsafe copying build/lib.macosx-10.6-intel-2.7/markupsafe/_speedups.c -> build/bdist.macosx-10.6-intel/wheel/markupsafe copying build/lib.macosx-10.6-intel-2.7/markupsafe/_speedups.so -> build/bdist.macosx-10.6-intel/wheel/markupsafe copying build/lib.macosx-10.6-intel-2.7/markupsafe/tests.py -> build/bdist.macosx-10.6-intel/wheel/markupsafe running install_egg_info Copying MarkupSafe.egg-info to build/bdist.macosx-10.6-intel/wheel/MarkupSafe-0.23-py2.7.egg-info running install_scripts creating build/bdist.macosx-10.6-intel/wheel/MarkupSafe-0.23.dist-info/WHEEL
!ls markupsafe/dist
MarkupSafe-0.23-cp27-none-macosx_10_6_intel.whl
Done! But what is this thing?
%%bash
mkdir wheel_contents
cd wheel_contents
unzip ../markupsafe/dist/MarkupSafe-0.23-cp27-none-macosx_10_6_intel.whl
Archive: ../markupsafe/dist/MarkupSafe-0.23-cp27-none-macosx_10_6_intel.whl inflating: markupsafe/__init__.py inflating: markupsafe/_compat.py inflating: markupsafe/_constants.py inflating: markupsafe/_native.py inflating: markupsafe/_speedups.c inflating: markupsafe/_speedups.so inflating: markupsafe/tests.py inflating: MarkupSafe-0.23.dist-info/DESCRIPTION.rst inflating: MarkupSafe-0.23.dist-info/metadata.json inflating: MarkupSafe-0.23.dist-info/top_level.txt inflating: MarkupSafe-0.23.dist-info/WHEEL inflating: MarkupSafe-0.23.dist-info/METADATA inflating: MarkupSafe-0.23.dist-info/RECORD
It seems too easy. Can I ship this wheel? Maybe it depends on some library on my system I don't know about?
%%bash
pip install delocate
Downloading/unpacking delocate Requirement already satisfied (use --upgrade to upgrade): wheel in /Users/mb312/.virtualenvs/lightning/lib/python2.7/site-packages (from delocate) Installing collected packages: delocate Successfully installed delocate Cleaning up...
!delocate-listdeps -a markupsafe/dist/MarkupSafe-0.23-cp27-none-macosx_10_6_intel.whl
/usr/lib/libSystem.B.dylib
No - looks good - go ahead and ship.
!delocate-listdeps -a scipy-0.14.0-cp27-none-macosx_10_6_intel.whl
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate /usr/lib/libSystem.B.dylib /usr/lib/libstdc++.6.0.9.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
%%bash
mkdir fixed_wheels
delocate-wheel -w fixed_wheels scipy-0.14.0-cp27-none-macosx_10_6_intel.whl
delocate-listdeps -a fixed_wheels/scipy-0.14.0-cp27-none-macosx_10_6_intel.whl
/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate /usr/lib/libSystem.B.dylib /usr/lib/libstdc++.6.0.9.dylib @loader_path/../../../../.dylibs/libgcc_s.1.dylib @loader_path/../../../../.dylibs/libgfortran.3.dylib @loader_path/../../../../.dylibs/libquadmath.0.dylib @loader_path/../../../.dylibs/libgcc_s.1.dylib @loader_path/../../../.dylibs/libgfortran.3.dylib @loader_path/../../../.dylibs/libquadmath.0.dylib @loader_path/../../.dylibs/libgcc_s.1.dylib @loader_path/../../.dylibs/libgfortran.3.dylib @loader_path/../../.dylibs/libquadmath.0.dylib @loader_path/../.dylibs/libgcc_s.1.dylib @loader_path/../.dylibs/libgfortran.3.dylib @loader_path/../.dylibs/libquadmath.0.dylib @loader_path/libgcc_s.1.dylib @loader_path/libquadmath.0.dylib
!pip wheel -w my_cache ipython[notebook]
Downloading/unpacking ipython[notebook] Saved ./my_cache/ipython-2.0.0-py27-none-any.whl Downloading/unpacking gnureadline (from ipython[notebook]) Saved ./my_cache/gnureadline-6.3.3-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl Downloading/unpacking jinja2 (from ipython[notebook]) Saved ./my_cache/Jinja2-2.7.2-py27-none-any.whl Downloading/unpacking pyzmq>=2.1.11 (from ipython[notebook]) Downloading pyzmq-14.3.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl (1.2MB): 1.2MB downloaded Saved ./my_cache/pyzmq-14.3.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl Downloading/unpacking tornado>=3.1 (from ipython[notebook]) Saved ./my_cache/tornado-3.2.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl Downloading/unpacking markupsafe (from jinja2->ipython[notebook]) Saved ./my_cache/MarkupSafe-0.23-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl Downloading/unpacking backports.ssl-match-hostname (from tornado>=3.1->ipython[notebook]) Saved ./my_cache/backports.ssl_match_hostname-3.4.0.2-py27-none-any.whl Cleaning up...
ls my_cache
Jinja2-2.7.2-py27-none-any.whl MarkupSafe-0.23-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl backports.ssl_match_hostname-3.4.0.2-py27-none-any.whl gnureadline-6.3.3-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl ipython-2.0.0-py27-none-any.whl pyzmq-14.3.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl tornado-3.2.1-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl
And if you do that for a few packages, and put them online, you can build installers for the whole scipy stack and more:
%%bash
virtualenv scipy-stack-lite
. scipy-stack-lite/bin/activate
pip install numpy scipy matplotlib
New python executable in scipy-stack-lite/bin/python Installing setuptools, pip...done. Downloading/unpacking numpy Downloading/unpacking scipy Downloading/unpacking matplotlib Downloading/unpacking nose (from matplotlib) Downloading/unpacking python-dateutil (from matplotlib) Running setup.py (path:/Users/mb312/talks/package_lightning/scipy-stack-lite/build/python-dateutil/setup.py) egg_info for package python-dateutil Downloading/unpacking tornado (from matplotlib) Downloading/unpacking pyparsing>=1.5.6 (from matplotlib) Running setup.py (path:/Users/mb312/talks/package_lightning/scipy-stack-lite/build/pyparsing/setup.py) egg_info for package pyparsing Downloading/unpacking six (from python-dateutil->matplotlib) Downloading six-1.6.1-py2.py3-none-any.whl Downloading/unpacking backports.ssl-match-hostname (from tornado->matplotlib) Installing collected packages: numpy, scipy, matplotlib, nose, python-dateutil, tornado, pyparsing, six, backports.ssl-match-hostname Running setup.py install for python-dateutil Running setup.py install for pyparsing Successfully installed numpy scipy matplotlib nose python-dateutil tornado pyparsing six backports.ssl-match-hostname Cleaning up...
!curl https://nipy.bic.berkeley.edu/scipy_installers/scipy-stack-1.0-plus.txt
# pip requirements file for scipy stack 1.0 # http://www.scipy.org/stackspec.html # # use with `pip install -r scipy-stack-1.0.txt` # pip requirement is not part of scipy-stack, but makes it possible to use OSX # wheels # numexpr is not part of scipy-stack but gives extra speed for pandas # ipython[test] is not part of the scipy-stack but allows us to test ipython -f https://nipy.bic.berkeley.edu/scipy_installers pip>=1.5.4 numpy>=1.6 scipy>=0.10 pytz tornado python-dateutil matplotlib>=1.1 pyzmq ipython[notebook,test]>=0.13 numexpr pandas>=0.8 sympy>=0.7 nose>=1.1
We can test it on a real machine. Here's a test run on an OSX 10.6 bare machine with no compiler, no Python.org Python. We install Python.org Python and the whole scipy / statistics / data stack via pip and wheels, and run tests on the installed packages. http://nipy.bic.berkeley.edu/builders/scipy-stack-2.7.6-wheel-requires/builds/11
The same machine, installing and testing on Python 3.3: http://nipy.bic.berkeley.edu/builders/scipy-stack-3.3.5-wheel-requires/builds/4
This runs every morning on our buildbot - configuration at https://github.com/nipy/nibotmi
Thanks to Matt Terry and travis-ci.org: https://travis-ci.org/matthew-brett/scipy-stack-osx-testing/builds/25131865
Testing wheels on bare OSX 10.9 virtual machines with Python from: the OSX system; Python.org; homebrew and macports, with and without virtualenvs.
Code at: https://github.com/matthew-brett/scipy-stack-osx-testing
Thank you packaging team. I think you cracked it.